{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正则表达式\n",
    "\n",
    "正则表达式（Regular Expression）是字符串处理的常用工具，通常被用来检索、替换那些符合某个模式（Pattern）的文本。很多程序设计语言都支持正则表达式，像Perl、Java、C/C++。在 Python 中是通过标准库中的re 模块 提供对正则的支持。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 正则表达式介绍\n",
    "在使用 re 模块之前，先来了解一下正则表达式的基本语法。\n",
    "\n",
    "### 用途\n",
    "通过使用正则表达式，可以：\n",
    "- 测试字符串内的模式。—— 例如，可以测试输入字符串，以查看字符串内是否出现电话号码模式或信用卡号码模式。这称为数据验证。\n",
    "- 替换文本。—— 可以使用正则表达式来识别文档中的特定文本，完全删除该文本或者用其他文本替换它。\n",
    "- 基于模式匹配从字符串中提取子字符串。—— 可以查找文档内或输入域内特定的文本。\n",
    "\n",
    "### 语法\n",
    "本文主要介绍正则的基本语法以及 re 模块的使用，不包括如何编写高效的正则表达式、如何优化正则表达式，这些主题请看其他教程。\n",
    "下图列出了Python支持的一些正则表达式元字符和语法：\n",
    "\n",
    "#### 普通字符\n",
    "\n",
    "普通字符包括没有显式指定为元字符的所有可打印和不可打印字符。这包括所有大写和小写字母、所有数字、所有标点符号和一些其他符号。\n",
    "\n",
    "#### 非打印字符\n",
    "\n",
    "非打印字符也可以是正则表达式的组成部分。下表列出了表示非打印字符的转义序列：\n",
    "\n",
    "| 字符  | 描述 |\n",
    "| ------------- |:-------------:| \n",
    "|\\cx\t| 匹配由x指明的控制字符。例如， \\cM 匹配一个 Control-M 或回车符。x 的值必须为 A-Z 或 a-z 之一。否则，将 c 视为一个原义的 'c' 字符。 |\n",
    "| \\f\t| 匹配一个换页符。等价于 \\x0c 和 \\cL。|\n",
    "| \\n\t| 匹配一个换行符。等价于 \\x0a 和 \\cJ。|\n",
    "|\\r\t| 匹配一个回车符。等价于 \\x0d 和 \\cM。 |\n",
    "|\\s\t| 匹配任何空白字符，包括空格、制表符、换页符等等。等价于 [ \\f\\n\\r\\t\\v]。|\n",
    "| \\S |\t匹配任何非空白字符。等价于 [^ \\f\\n\\r\\t\\v]。|\n",
    "| \\t\t| 匹配一个制表符。等价于 \\x09 和 \\cI。|\n",
    "|\\v\t| 匹配一个垂直制表符。等价于 \\x0b 和 \\cK。|\n",
    "\n",
    "\n",
    "#### 特殊字符\n",
    "\n",
    "| 特殊字符  | 描述 |\n",
    "| ------------- |:-------------:| \n",
    "| \\$\t| 匹配输入字符串的结尾位置。如果设置了 RegExp 对象的 Multiline 属性，则 $ 也匹配 '\\n' 或 '\\r'。要匹配 $ 字符本身，请使用 \\$。|\n",
    "|( )\t| 标记一个子表达式的开始和结束位置。子表达式可以获取供以后使用。要匹配这些字符，请使用 \\( 和 \\)。|\n",
    "|*\t|匹配前面的子表达式零次或多次。要匹配 * 字符，请使用 \\*。|\n",
    "|+\t|匹配前面的子表达式一次或多次。要匹配 + 字符，请使用 \\+。|\n",
    "|.\t|匹配除换行符 \\n之外的任何单字符。要匹配 .，请使用 \\。|\n",
    "|[\t|标记一个中括号表达式的开始。要匹配 [，请使用 \\[。|\n",
    "|?\t| 匹配前面的子表达式零次或一次，或指明一个非贪婪限定符。要匹配 ? 字符，请使用 \\?。 |\n",
    "| \\\t| 将下一个字符标记为或特殊字符、或原义字符、或向后引用、或八进制转义符。例如， 'n' 匹配字符 'n'。'\\n' 匹配换行符。序列 '\\\\' 匹配 \"\\\"，而 '\\(' 则匹配 \"(\"。|\n",
    "| ^ |\t匹配输入字符串的开始位置，除非在方括号表达式中使用，此时它表示不接受该字符集合。要匹配 ^ 字符本身，请使用 \\^。 |\n",
    "|{\t| 标记限定符表达式的开始。要匹配 {，请使用 \\{。|\n",
    "| &#124; | 指明两项之间的一个选择。要匹配 &#124;，请使用 \\ &#124;。|\n",
    "\n",
    "\n",
    "\n",
    "#### 限定符\n",
    "限定符用来指定正则表达式的一个给定组件必须要出现多少次才能满足匹配。有*或+或?或{n}或{n,}或{n,m}共6种。\n",
    "正则表达式的限定符有：\n",
    "\n",
    "| 特殊字符  | 描述 |\n",
    "| ------------- |:-------------:| \n",
    "| \\* | 对于前一个字符重复0到无穷次| | \n",
    "| \\+ | 对于前一个字符重复1到无穷次| \n",
    "| ？| 对于前一个字符重复0到1次| \n",
    "| {m,n} | 对于前一个字符重复次数在为m到n次，其中，{0,} = *,{1,} = , {0,1} = ?| \n",
    "| {m} | 对于前一个字符重复m次| \n",
    "\n",
    "\n",
    "#### 定位符\n",
    "定位符使您能够将正则表达式固定到行首或行尾。它们还使您能够创建这样的正则表达式，这些正则表达式出现在一个单词内、在一个单词的开头或者一个单词的结尾。\n",
    "定位符用来描述字符串或单词的边界，^和$分别指字符串的开始与结束，\\b描述单词的前或后边界，\\B表示非单词边界。\n",
    "正则表达式的限定符有：\n",
    "\n",
    "| 字符  | 描述 |\n",
    "| ------------- |:-------------:| \n",
    "| ^ | 匹配字符串开头，在 Multiline模式下，还会匹配每一行的开头 |\n",
    "| $ | 匹配字符串末尾， 在Multiline模式下，还会匹配每一行的末尾 |\n",
    "| \\\\A |  仅匹配字符串开头，例如“\\Axyz”匹配“xyz   ” |\n",
    "| \\\\Z | 仅匹配字符串末尾，例如“abc\\Z”匹配“   xyz” |\n",
    "| \\\\b | 匹配一个单词的边界，即单词与空格的位置，例如，“er\\b\"，匹配“never”中的er，但不匹配“verb”中的“er”|\n",
    "| \\\\B | 非单词边界，“er\\B\"，匹配“verb”中的“er”，但不匹配“never”中的“er” |\n",
    "\n",
    "注意：不能将限定符与定位点一起使用。由于在紧靠换行或者字边界的前面或后面不能有一个以上位置，因此不允许诸如 ^* 之类的表达式。\n",
    "若要匹配一行文本开始处的文本，请在正则表达式的开始使用 ^ 字符。不要将 ^ 的这种用法与中括号表达式内的用法混淆。\n",
    "若要匹配一行文本的结束处的文本，请在正则表达式的结束处使用 $ 字符。\n",
    "\n",
    "#### 预定义字符集\n",
    "\n",
    "| 字符  | 描述 |\n",
    "| ------------- |:-------------:| \n",
    "| \\d | 匹配任何十进制数；它相当于类 [0-9]。|\n",
    "| \\D | 匹配任何非数字字符；它相当于类 [^0-9]。|\n",
    "| \\s | 匹配任何空白字符；它相当于类 [ fv]。|\n",
    "| \\S | 匹配任何非空白字符；它相当于类 [^ fv]。|\n",
    "| \\w | 匹配任何字母数字字符；它相当于类 [a-zA-Z0-9_]。|\n",
    "| \\W | 匹配任何非字母数字字符；它相当于类 [^a-zA-Z0-9_]。|\n",
    "\n",
    "\n",
    "\n",
    "### 贪婪模式与非贪婪模式\n",
    "“贪婪模式”总是尝试匹配尽可能多的字符；“非贪婪模式”则相反，总是匹配尽可能少的字符。例如，用\"ab*\"如果用于查找\"abbbc\"，将找到\"abbb\"。而如果使用非贪婪的数量词\"ab*?\"，将找到\"a\"。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  使用 re 模块\n",
    "\n",
    "<http://jingyan.baidu.com/article/f54ae2fcd215b21e92b849f3.html><Reference>\n",
    "\n",
    "<http://blog.csdn.net/lisonglisonglisong/article/details/39697137><Python3标准库（二） re模块>\n",
    "\n",
    "### 编译正则表达式\n",
    "re 模块提供了 re.compile() 函数将一个字符串编译成 pattern object，用于匹配或搜索。函数原型如下：\n",
    "\n",
    "    re.compile(pattern, flags=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "re.compile(r\"abc\")  # 创建pattern对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "text = \"The quick brown fox jumped over the lazy dog\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match = re.search('quick', text)\n",
    "match.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.end()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None"
     ]
    }
   ],
   "source": [
    "match = re.match('quick', text)\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object at 0x247b718>"
     ]
    }
   ],
   "source": [
    "match = re.match('.*quick', text)\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<_sre.SRE_Match at 0x2486030>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match = re.search('brown ([a-z]+) jumped', text)\n",
    "match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('fox',)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<_sre.SRE_Match at 0x2486120>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match = re.search('brown ([a-z]+)', text)\n",
    "match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('fox',)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<_sre.SRE_Match at 0x2486198>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match = re.search('brown ([a-z]+?)', text)\n",
    "match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('f',)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('bar baz',)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = 'foo bar baz baz'\n",
    "re.search('foo (.*) baz', s).groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('bar',)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = 'foo bar baz baz'\n",
    "re.search('foo (.*?) baz', s).groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('fox',)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = re.search('brown (?P<animal>[a-z]+) jumped', text)\n",
    "m.groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'animal': 'fox'}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.groupdict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "re_time = re.compile('(\\\\d{2}):(\\\\d{2}):(\\\\d{2})')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('01', '23', '45')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re_time.match('01:23:45').groups()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can make regex strings more readable by using 'raw strings':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "re_time = re.compile(r'(\\d{2}):(\\d{2}):(\\d{2})')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('01', '23', '45')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re_time.match('01:23:45').groups()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use 'verbose mode':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "re_time = re.compile(r'''\n",
    "(\\d{2}) # hour\n",
    ":\n",
    "(\\d{2}) # minute\n",
    ":\n",
    "(\\d{2}) # second\n",
    "''', re.VERBOSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('01', '23', '45')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re_time.match('01:23:45').groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text = '''The quick\n",
    "brown fox\n",
    "jumped over\n",
    "the lazy dog'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None"
     ]
    }
   ],
   "source": [
    "print(re.search('quick brown', text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object at 0x2486468>"
     ]
    }
   ],
   "source": [
    "print re.search('quick( |\\n)brown', text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None"
     ]
    }
   ],
   "source": [
    "print(re.search('quick.brown', text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object at 0x247b718>"
     ]
    }
   ],
   "source": [
    "print(re.search('quick.brown', text, re.DOTALL))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The\n",
      "quick\n",
      "brown\n",
      "fox\n",
      "jumped\n",
      "over\n",
      "the\n",
      "lazy\n",
      "dog"
     ]
    }
   ],
   "source": [
    "for m in re.finditer('(\\w+)', text): \n",
    "    print(m.group(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slow\n",
      "brown fox\n",
      "jumped over\n",
      "the lazy dog"
     ]
    }
   ],
   "source": [
    "print(re.sub('quick', 'slow', text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 5\n",
      "5 3\n",
      "6 4\n",
      "3 4 3"
     ]
    }
   ],
   "source": [
    "def sub_length(match):\n",
    "    return str(len(match.group(1)))\n",
    "\n",
    "print(re.sub('(\\w+)', sub_length, text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 5\n",
      "5 3\n",
      "jumped over\n",
      "the lazy dog"
     ]
    }
   ],
   "source": [
    "print(re.sub('(\\w+)', sub_length, text, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "large_text = open('Regular Expressions.ipynb').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "regular_split = large_text.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['{',\n",
       " '\"metadata\":',\n",
       " '{',\n",
       " '\"name\":',\n",
       " '\"Regular',\n",
       " 'Expressions\"',\n",
       " '},',\n",
       " '\"nbformat\":',\n",
       " '2,',\n",
       " '\"worksheets\":']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regular_split[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['',\n",
       " 'metadata',\n",
       " 'name',\n",
       " 'Regular',\n",
       " 'Expressions',\n",
       " 'nbformat',\n",
       " '2',\n",
       " 'worksheets',\n",
       " 'cells',\n",
       " 'cell_type']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "better_split = re.split('\\W*', large_text)\n",
    "better_split[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "- Write a function that finds all integers in a file using\n",
    "  regular expressions\n",
    "- Write a function that finds all capitalized words in a file\n",
    "- Write a function that replaces all instances of `'<br>'` in a \n",
    "  file with `'<br/>'`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
